test: add colocated core component acceptance suite - #841

Map the 131 merged PRs between e2cf39a and 0fb3b24 to their
acceptance regressions, list the product findings, point the README at
the new audit and update the case and configuration field counts.

Signed-off-by: Kevin Wang <wy721@qq.com>

Signed-off-by: Kevin Wang <wy721@qq.com>

start-simulator.sh launches the simulator through `setsid sg`, and sg
forks, so the recorded pid is the sg group leader rather than the
simulator. stop-simulator.sh only polled that pid: TERM ended sg at once
and the helper went on to `find -delete` the runtime while the simulator
was still unlinking dstack.sock and its lock. The sweep reported
tc-gos-setup-002 as ERROR over a PASS result with

  simulator fixture cleanup failed: find: cannot delete
  '.../dstack.sock.lock': No such file or directory

Poll the process group instead, and wait for it after KILL as well.

Signed-off-by: Kevin Wang <wy721@qq.com>

tc-gos-concurrency-002 raised out of head_of_line() before returning
its observations, so a failed run kept only the assertion text: no
quote cost, no probe latencies, no agent (pid, start time) and no
journal. The 0924 physical-TDX sweep failed with 5/48 GetQuote calls
reset and left nothing to tell a restart from a transport drop.

Record the offset, duration and error of every failed saturating call,
fetch the agent journal whenever a call or probe failed, and carry the
observations into the step record when an expectation fails.

Signed-off-by: Kevin Wang <wy721@qq.com>

tc-kms-runtime-004 runs the mock RPC by cutting the server out of the
runtime-001 helper and prepending its own two-line header. 1956cb7
gave that server a `record_call` which reads `os.environ`, and the header
only imported `http.server` and `json`, so every eth_* call raised
NameError inside the container. auth-api never saw a chain, health stayed
at "authorization backend unavailable", and wait_health timed out.

That exception also left main() before result.json was written, so the
sweep reported ERROR with no evidence, and none of the retained
containers were named anywhere. Catch it as the case's failure instead:
the result is FAIL with the error in the summary and the retained Docker
resources recorded in the artifact.

Signed-off-by: Kevin Wang <wy721@qq.com>

…restart

tc-gos-platform-006 step 3 treated dstack-guest-agent as a leaf: it ran
`systemctl restart dstack-guest-agent` and gave the Tappd route 45 s to
come back. Since #1324, app-compose.service has
Requires=dstack-guest-agent.service, so systemd propagates the restart
to it. Its ExecStop stops the fixture bridge that carries the Tappd
route, and its ExecStart reruns the fixture's pre-launch `docker load`
before recreating the bridge. On an idle tdxlab host the guest journal
shows app-compose stopped at +0 s, the agent back at +1 s, and the
containers started at +26 s; the route recovered 30.4 s after the
restart. Under the 4-worker sweep the load took longer and every probe
in the 45 s window was reset (ConnectionResetError).

Wait up to 240 s for app-compose.service to be active with no job
queued before repeating the RPC, record whether it was restarted with
the agent and how long recovery took, and document the propagation in
the case.

Signed-off-by: Kevin Wang <wy721@qq.com>

tc-gos-concurrency-002 opened all 24 GetQuote connections at once. On
physical TDX the DstackGuest listener is reached through a QEMU
user-networking port forward, and libslirp listens on the host side
with a backlog of one (`ss -ltn` shows Send-Q 1 on a hostfwd socket).
Under the 4-worker sweep, 5/48 calls were reset.

Rerunning three leases in parallel reproduced it: one run reset 2 calls,
both issued in the first 13 ms of the burst and reset after 1.0 s and
3.3 s, which is the SYN retransmit schedule. The agent kept its pid and
start time with NRestarts=0, its journal had no warnings, and the bridge
container had not restarted. The guest agent did not drop them.

Start the loaders 50 ms apart. One quote costs about 1.02 s and quotes
run one at a time, so all 24 callers still queue on the quote lock:
load_calls stays at 43 per run. With the ramp, 8 of 8 runs passed in
two rounds of 4 parallel leases.

Signed-off-by: Kevin Wang <wy721@qq.com>

tc-gos-observabil-004 writes 128 MiB to the first data disk and requires
SysInfo's free_size to drop. It failed with disk_free_space_decreased in
the 0924 4-worker sweep and once before (fcb8bfc), and passed on
rerun both times. The disk is the ZFS dataset dstack/data, and the
fixture's app-compose run ends with `docker image prune -af`, which
reclaims about 570 MB of images its pre-launch script loaded. The
baseline was taken without syncing the pool, so frees still pending
from that prune could outgrow the write.

Before the baseline, wait for app-compose.service to be active, run
`zpool sync`, and wait for the pool's `freeing` to reach zero, bounded
to fit the 90 s SSH timeout. Record the free-space delta in the artifact
so the next failure shows its size.

Not reproduced: 4 unpatched runs in parallel passed with freeing=0 at
the baseline and a delta of -128/-129 MiB. 4 patched runs in parallel
passed with the same delta.

Signed-off-by: Kevin Wang <wy721@qq.com>

The measure CLI defaulted --hotplug-off to false and now follows the VMM
default of true (PR #1405), while diagnose reads hotplug_off from the
VmConfig, where it defaults to false. The diagnose-matches-measure row
therefore compared two machines that differ in the ACPI tables, and
RTMR[0] diverged.

Pass --hotplug-off on every measure row and hotplug_off in the diagnose
VmConfig, so the matrix measures the same machine whichever default the
CLI ships. The advanced-machine-fields row now turns hotplug on so it
still moves the field away from the baseline.

Signed-off-by: Kevin Wang <wy721@qq.com>

…nc state

The health/exit case waited for the restarted node's RPC port and then
called Debug.GetSyncData once. The main, admin, and debug listeners bind
independently, so under load the debug port was not up yet, the call
failed at the transport (sync=None), and the case reported the persisted
instance as lost (retained=0). Retry the read for up to ten seconds.

Signed-off-by: Kevin Wang <wy721@qq.com>

StopVm returns once the supervisor has been told to kill QEMU, not once
QEMU has exited. A StartVm that lands in that window sees the VM still
running and launches nothing, so the guest stays down until the VMM's
exited-VM check restarts it about 30s later. Under a loaded sweep that
detour pushed the reboot past the 150s recovery budget. Poll VMM info
until the VM is no longer running, as tc-gos-setup-007 already does.

Signed-off-by: Kevin Wang <wy721@qq.com>

tc-gos-yocto-004 had a 60s case timeout while it gives the in-guest
lifecycle script 300s. Recorded runs took 44-58s and timed out at 60s in
four sweeps, all under concurrent load. Raise the case timeout to 360s so
the runner no longer cuts the script short.

Signed-off-by: Kevin Wang <wy721@qq.com>

…oint

The corrupt-layer row flipped the byte at the middle of the eStargz blob
and expected the lazy pull or the read of /integrity-marker to fail. A
lazy mount only reads a file through its TOC entry, so when the midpoint
fell in the prefetch landmark or a tar header member nothing ever read
it: the pull and the read both succeeded and the row failed on an
image the snapshotter handled correctly.

Locate the marker's payload member from the TOC and flip the first
deflate byte that changes the bytes a read returns. The read now fails
with EIO on every run.

Signed-off-by: Kevin Wang <wy721@qq.com>

Every KMS upgrade matrix case runs its own VMM over the same host port
range, and free_ports only excluded ports that its own VMM's VMs
forward. A port whose VM was stopped, or whose QEMU had not bound it
yet, looked free to a concurrent case. Whichever QEMU bound first won,
so under four workers a client observer answered with another case's
KMS (a Rocket 404), and an in-place upgraded Gateway restarted onto a
port that no longer reached it (TLS listener timeout).

Record each allocated port under the shared state root, named after
the owning case workspace, and skip ports whose owner is still alive.
The provider removes the workspace on lease release, which frees the
marker for reuse.

Signed-off-by: Kevin Wang <wy721@qq.com>

The in-place Gateway upgrade check matched wavekv v1's "Node status
after bootstrap" line, which only the v0.5.11 binary prints; the
candidate uses wavekv 2 and never logs it. The line showed up only when
dockerd restarted the stopped legacy container on boot, before compose
recreated it from the candidate image, so the check passed or failed on
that race, and the TLS readiness probe could also be answered by the
legacy container.

Wait for the candidate's own WaveKV load summary, require it to report
the baseline client's instance with no unreadable rows, and only then
probe the candidate listener.

Signed-off-by: Kevin Wang <wy721@qq.com>

#1415 reverts #1331, so the image again ships dstack-image.conf in
tmpfiles.d and the first-boot unit. tc-gos-platform-005 keeps the
outcome that matters, a root-owned 0755 TPM keystore at runtime, and the
source catalogs follow the file back to its old path.

Signed-off-by: Kevin Wang <wy721@qq.com>

Signed-off-by: Kevin Wang <wy721@qq.com>